Systems and methods of creating a three-dimensional virtual image

ABSTRACT

Embodiments of the present invention create a three-dimensional virtual model by a user identifying a three-dimensional object, capturing a plurality of two-dimensional images of said object in succession, said plurality of images being captured from different orientations, recording said plurality of images on a storage medium, determining the relative change in position of said plurality of images by comparing two subsequent images, wherein the relative change is determined by a difference in color intensity values between the pixels of one image and another image, generating a plurality of arrays from the difference determined and generating a computer image from said plurality of arrays, wherein said computer image represents said three-dimensional object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/219,583, filed Sep. 16, 2015 the entire contentsof which are hereby incorporated herein by reference for all that itteaches and for all purposes.

FIELD OF THE INVENTION

The systems and methods relate to creating a three-dimensional virtualimage, model, and using a single image capture device, such as a digitalcamera.

BACKGROUND

Often times in production and design, it is useful to generate small,scaled models of a subject product or building prior to final productfabrication or final build. These models provide a concrete, visualobject that pulls all conceptual elements into one object, and thereforemake it more reasonable to consider the product or building as a whole.By considering the model as a whole, criticisms can be discovered andrectified before an object, such as the product or building itselfand/or a component thereof, is produced, and can therefore increaseefficiency and reduce costs during the production process.

Though tremendously invaluable, generating a useful model is a verytime-consuming process, and one that requires exceptional skill. Becausesuch skill is required, producing a model typically necessitates theemployment of a well-trained artist, which can be very expensive.Because of the highly customized nature of models, each additional modelwill incur a similar expense, even if models are nearly identical.Further, although they are cheaper to produce than full-scaleprototypes, models often require highly specialized building materials.Because of these requirements, models can still be quite expensive tocreate.

One alternative to traditional modeling is to create a virtual model. Avirtual model is a computational representation of a model in which adigital coordinate system (such as a Cartesian system) represents athree-dimensional (3D) space of the real world, and coordinates for manypoints that are displayed within this system such that an object appearson a monitor, screen, display, or other output device similar to itsreal world counterpart. The number of points representing the object inquestion affects its fidelity: more points create a more accuraterepresentation, although greater numbers of points require greatercomputational power.

Virtual models are a valuable resource in production and design, andhave supplemented or even replaced more conventional models andarchitectural renderings outright in some applications. Virtual modelsoffer designers a way to generate models of objects that can be easilymodified; size, shape, color, and many other physical aspects can bemodified in a relatively easy manner. Each change can be renderedwithout the use of any new physical material, and do not require theproduction of a new physical product, which would be very expensive andtime consuming. Virtual models make it possible to visualize the productin question in its intended setting, such as a rendering of a buildingthat does not yet exist in an existing lot. Further, undesirable changescan be quickly and easily undone, and multiple instances of virtualmodels can be viewed simultaneously for comparison. Virtual, interactiverenderings can instead be generated digitally on a computer monitor orother display, and/or sent to a three-dimensional printer for cheap,fast prototyping, making virtual models a tremendously efficient qualitycontrol step in the production process, or even a complete alternativeto traditional prototyping.

One very large advantage of digital/virtual models is the ability toinstantly create exact replicas. Duplicated models can be easilymodified for subtle changes and compared for aesthetic or any otherpreferences. In traditional modeling, each model requires substantialtime and resources, and exact replicas are not possible because ofsubtle variations in the model-making process. In most cases, anysubstantial changes to a model that are desired must be consideredbefore construction of a new instance of the model in order to beincorporated. This limitation makes it more difficult to affect furthermodifications that are conceived of during model making, which cantherefore be a detriment to the creative process. Changes that can bemade to an existing model after construction necessarily replace theoriginal form, making side-by-side comparison impossible. Both of theselimitations are easily navigated by the use of virtual models. Changesare easily incorporated at any stage of model development, and multiple,exact replicas can be instantaneously generated and quickly modified forside-by-side comparison or any other purpose.

While virtual representations are an advantageous tool for producers ofcreative content, the generation of such virtual content is highlyspecialized. The ability to generate such a useful model is highlydependent upon the skill of the modeler, and upon the modeler'sfamiliarity with software that facilitates such modeling. Such softwareis highly technical and can take years to learn. Because of its highlyspecialized application, acquiring this skill is often unreasonable forinventors from varied backgrounds. Furthermore, such software is oftenprohibitively expensive. In short, the software required to generatevirtual content itself, along with the ability to use it, are themselvesoften a barrier to efficiently generating useful models.

One particularly useful strategy in navigating technical challenges isto generate three-dimensional models based on existing objects in thereal, physical world and modify them. Such models provide a detailedsubstrate for future modification, thus reducing the overall designburden. While useful, generating three-dimensional content based on realworld objects is currently also challenging and time consuming for manymodelers. This makes it impractical for many designers, despiteregularly needing to do so for the purposes of either generatingstarting forms upon which a new model will be based, or as secondaryobjects that will interact with the model in some way. Modelers that doattempt to model real world objects are confined to the immobile natureof modeling software. Rather than modeling an object as they can see itin the real world, a modeler must find adequate images of the objectthey wish to model from many views. Typically, this requires that ablueprint of the object, detailing many views of the object in question,is available to them. Such blueprints are themselves highly technical,time consuming, and often expensive to generate.

One solution to these challenges is automated, imaging-based softwarecapable of analyzing multiple two-dimensional images of the same objectand generating three-dimensional coordinates of the object from theseimages. This would eliminate the technical challenges associated withgenerating such a model, the cost barrier of expensive modelingsoftware, and the requirement of existing images or blueprints. Once thedifficult part of generating a three-dimensional shape has been carriedout, a designer can then modify the shape quickly and with relativelylittle training. Several three-dimensional scanned objects (or parts ofobjects) could then be assembled into a final model with relative ease.

One example of prior art has attempted to solve this problem usinglasers and rotating platters. In this system, an object is placed on therotating platter, and one or more lasers are aimed at the object. Thedistance from the laser to the object changes over time as the objectrotates and the laser moves across the surface of the object, and thesedistances are detected and stored as a series of range vectors. Theserange vectors can be used to reconstruct the surface of the object invirtual space, thus producing a virtual model of the object in question.This prior art requires hardware that is expensive and inaccessible tomost designers. Furthermore, such a device is immobile, and limited toobjects small enough to fit on the rotating platter

Another example of an existing product is a piece of hardware consistingof a special camera that attaches to a tablet. The hardware is used toproject an infrared laser beam in a specific pattern. The camera mountedon the tablet can then be used to “scan” back and forth across an objectby way of user movement. An infrared sensor on the device collects andrecords distortions in a pattern at VGA resolution while scanning. Thedata collected from this process can then be used to create a depth mapof the material being scanned. The tablet's camera is also invoked torecord color information. The data can then be exported to software,which reassembles the images into a three-dimensional model.

The prior art requires special hardware and a secondary export tosoftware, which requires additional resources such as a user's time andan expense of additional equipment, such as requiring a user to carryadditional hardware. The additional hardware also requires its ownbattery and must be charged in addition to the tablet. Further, theextra hardware requirement limits the context in which the device isused. Specifically, its use is limited to tablets such as iPads, whichrequires a user to purchase one if they do not own one, in addition tothe costs of specialized hardware. Any future use with anything otherthan a tablet would require new hardware to be developed in order tointerface with existing hardware, such as a satellite, microscope, ormobile phone.

Another example of an existing product uses the camera in a mobile phoneto acquire images where a user places an object on a reference surfaceand walks around the object, taking many static images of the objectfrom many different angles, and revolving the object several times inthe process. Although this product does not require special hardware, itis limited with regard to producing the type of models to be used in aproduction environment. The process is also highly dependent upon theskill of the user in finding appropriate angles and lighting conditions,and in taking enough images. Because the software involves simply imageanalysis, the software is incapable of distinguishing many objects fromone another, and further requires the user to clean up the image byremoving elements of the image that were inappropriately included in thethree-dimensional model. This product is based on an existing ability tomap virtual three-dimensional meshes onto points of light and dark, andis meant to be used in the context of specialized software in order tobe useful, which is itself an example of the expensive modeling softwarementioned previously.

Another example of an existing product can be used on a standard mobilephone to record images, but must be subsequently processed on far morepowerful hardware. This requires a user to purchase very expensiveaccelerated graphics processors, or to pay a developer for access totheir hardware as a service. These additional cost barriers areostensibly the result of inefficient processing algorithms that requirevery powerful processors.

A process that does not require expensive hardware, software, orextensive training and experience would be far more desirable, and wouldsubstantially increase the accessibility of virtual model making. Theincreased accessibility and reduction on time and money demands couldnot only have a tremendous economic effect with the ability to generateprototypes of ideas, but should also help to identify problems anderrors more quickly. It would provide designers with a much faster wayof navigating the technical, difficult portion of model generation bytaking advantage of existing technology to automatically generate ashape that is imaged with a camera, and without requiring the purchaseof any additional hardware.

Virtual models of real objects simulated in three-dimensional spacerendered in the manner described above have many possible applicationsbeyond production and design. Such applications can range from routineand hobbyist to highly technical, specialized use if the ability togenerate and use virtual models is more portable, requires lesstechnical training to use, and more financially accessible to morepeople. Interior design applications, for example, could be tremendouslyexpedited if one were able to portably record the interior of a room andhave the recording translated into a virtual model. Alternatively,scientists studying in the field could record virtual models ofspecimens for future analysis without disturbing the specimen or itshabitat. Applications such as this could not only save tremendously oncosts, but would allow work to be carried out in a non-invasive manner.

SUMMARY

It is with respect to the above issues and other problems that theembodiments presented herein were contemplated. That is, in accordancewith embodiments of the present disclosure, systems and methods arepresented for scanning virtually any real-world object with a singleimage capture device, such as a digital camera. In accordance withembodiments of the present disclosure, a user, without using multiplecameras, takes one image of an object, rotates the camera at some angleand then takes another image of the object. By merging the two imagestogether and then measuring a rotation, a depth of a scanned object canbe determined. A distance as a value may be provided in accordance withembodiments of the present disclosure. For example, a distance from amobile device with a camera to an object may be provided and laterutilized to create and/or modify a virtual three-dimensional model ofthe object.

Embodiments of the present disclosure may be used in existing mobiletechnology like smartphones, or any digital camera with some type ofrotational measurement device. A scanned object may be used for rapidprototyping and three-dimensional printing. Similar to a flat-bedtwo-dimensional scanner (e.g., a photocopier), embodiments of thepresent disclosure may scan any three-dimensional object and display avirtual three-dimensional object on a display of a smartphone and/orthree-dimensionally print the scanned object. For example, virtualcoordinates of an object may be sent straight to a 3D printer forphysical object duplication. The applications of such a technology maybe applied in many fields. For instance, if applied in the medicalfield, three-dimensional body scans may be obtained for surgical orother medical purposes. Such technology might also find tremendousutility in health applications. If, for example, a doctor could quicklygenerate a virtual model of some region on a patient in need of amedical procedure such as surgery, that doctor may familiarize herselfwith the region in question and get a sense of what must be done priorto surgery. This reduces physiological stress on the patient byshortening the duration of surgery and reduces the chances of accidentby allowing the physician to know exactly what to expect, allowing herto become mentally prepared for the operation. Changes to a virtualmodel may also be verified by the patient prior to the surgery, furtherreducing risk of accident.

Similarly, the entertainment industry consisting of movies and games forexample, may employ three-dimensional graphics on a regular basis,objects and characters may quickly be scanned and modeled. For example,video game developers or users may scan an object and create their ownelements to incorporate into video games. In accordance with embodimentsof the present disclosure, the security industry may benefit fromconducting body scans of assailants without the assailant's knowledgebecause, unlike a laser, a camera may be utilized as a passive measuringdevice, meaning, the scanned object would have no knowledge that it isbeing scanned. Moreover, security cameras may also reconstruct virtualmodels of an object and, if retrofitted, could be made to include dataoutside the visual spectrum of light, such as from X-ray or ultrasound,for example.

In accordance with embodiments of the present disclosure, otherapplications such as but limited to satellite imaging may find greatutility by employing systems, methods, and/or techniques describedherein. For example, various sized regions (e.g., small, medium, large)of Earth may be surveyed by satellite utilizing the methods and systemsdescribed herein; accordingly, three-dimensional virtual models may begenerated quickly and/or efficiently. Such virtual three-dimensionalmodels may be very useful for climate modeling, as one example. Suchsatellite imaging cameras may be used to reconstruct virtual models ofan object and, if retrofitted, could be made to include data outside thevisual spectrum of light, such as from X-ray or ultrasound, for example.

Likewise, artists seeking to use a real, physical object in somecapacity—such as for demolition in a film sequence—must currently eitherbuild a real, physical model of the item in question or generate avirtual one. The latter process requires extensive experience in virtualmodeling and access to expensive modeling computer programs, aspreviously mentioned. The process of generating a virtual model is timeconsuming for the modeler and thus can be very expensive for theproducer, all of which could be bypassed with a better imaging-basedsolution. Such a technology described herein may open up manypossibilities for use with hobbyists who simply like to collect, model,and manipulate virtual counterparts of real world objects and furtherexplore new areas of pastimes using such available technology. Suchexploration has potential for generating new uses and new industries.For example, vendors could generate libraries of catalogued items forexchange, charging premiums on rare and high fidelity items. Atechnology that overcomes the challenges previously mentioned wouldtherefore be tremendously invaluable to society.

In accordance with at least some embodiment of the present disclosure, amobile device is provided, the mobile device including a display, aprocessor, and one or more memories including one or more instructionsthat when executed by the processor, cause the processor to obtain aplurality of two-dimensional images of an object in succession, theplurality of two-dimensional images being captured form differentorientations, store said plurality of two-dimensional images into theone or more memories, determine a relative change in position of saidplurality of two-dimensional images by comparing two subsequent imagesof the plurality of two-dimensional images, wherein the relative changeis determined by a difference in color intensity values between pixelsof one of the two subsequent images and the other image, generate athree-dimensional coordinate based on the relative change in position,generate a computer image including the three-dimensional coordinate,wherein said computer image represents the three-dimensional object, andoutput to the display, the computer image representing thethree-dimensional object.

In accordance with at least some embodiments of the present disclosure,a method of generating a virtual three-dimension model of an object isprovided, the method including obtaining a plurality of two-dimensionalimages of the object in succession, the plurality of two-dimensionalimages being captured form different orientations, storing saidplurality of two-dimensional images into one or more memories,determining a relative change in position of said plurality oftwo-dimensional images by comparing two subsequent images of theplurality of two-dimensional images, wherein the relative change isdetermined by a difference in color intensity values between pixels ofone of the two subsequent images and the other image, generating athree-dimensional coordinate based on the relative change in position,and generating a computer image including the three-dimensionalcoordinate, wherein said computer image represents the three-dimensionalobject.

Further yet, and in accordance with at least some embodiment of thepresent disclosure, a non-transitory computer readable medium havingstored thereon instructions that cause a processor to execute a methodis provided, the method comprising one or more instructions to obtain aplurality of two-dimensional images of an object in succession, theplurality of two-dimensional images being captured form differentorientations, one or more instructions to store said plurality oftwo-dimensional images into the one or more memories, one or moreinstructions to determine a relative change in position of saidplurality of two-dimensional images by comparing two subsequent imagesof the plurality of two-dimensional images, wherein the relative changeis determined by a difference in color intensity values between pixelsof one of the two subsequent images and the other image, one or moreinstructions to generate a three-dimensional coordinate based on therelative change in position, one or more instructions to generate acomputer image including the three-dimensional coordinate, wherein saidcomputer image represents the three-dimensional object, and one or moreinstructions to output to the display, the computer image representingthe three-dimensional object.

Accordingly, it is these and other advantages that will be apparent fromthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 depicts a first system for creating a three-dimensional virtualimage in accordance with at least some embodiments of the presentdisclosure;

FIG. 2 depicts additional details of a communication device inaccordance with embodiments of the present disclosure;

FIGS. 3A-3B generally depict additional details with respect to a pixelmapping process in accordance with embodiments of the presentdisclosure;

FIG. 4 is a reference frame and associated triangle utilized inaccordance with embodiments of the present disclosure;

FIG. 5 depicts a flow diagram of a method generally directed to creatinga three-dimensional virtual image in accordance with embodiments of thepresent disclosure;

FIG. 6 depicts a flow diagram of a method generally directed tocapturing images in accordance with embodiments of the presentdisclosure;

FIG. 7 depicts a flow diagram of a method generally directed todetermining a three-dimensional reference point and determiningthree-dimensional coordinate points for an object of interest based onthe three-dimensional reference point in accordance with embodiments ofthe present disclosure;

FIG. 8 depicts a flow diagram of a method generally directed to a costfunction in accordance with embodiments of the present disclosure;

FIG. 9 depicts a second system for creating a three-dimensional virtualimage in accordance with at least some embodiments of the presentdisclosure;

FIG. 10 depicts a display of a communication device in accordance withat least some embodiments of the present disclosure; and

FIG. 11 generally depicts a visual representation of an iterativeestimation of distance based on a rate of movement of an object inaccordance with at least some embodiments of the present disclosure.

DETAILED DESCRIPTION

The ensuing description provides embodiments only, and is not intendedto limit the scope, applicability, or configuration of the claims.Rather, the ensuing description will provide those skilled in the artwith an enabling description for implementing the embodiments. It beingunderstood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope ofthe appended claims. To avoid unnecessarily obscuring the embodimentsdescribed herein, the following description omits well-known structures,components, and devices that may be shown in block diagram form or areotherwise summarized.

Similar to the way a human brain uses two images (one generated by eacheye) to estimate depth, in accordance with embodiments of the presentdisclosure, at least one embodiment estimates distance to an object orobjects of interest. For example, instead of two images separated inspace, certain embodiments instead use two subsequent images separatedby a small time delay and a measurable distance. For instance, a usermay capture an image of an object or objects of interest and then rotatean image capture device by some angle and capture another image. The twoimages may then be merged together and the rotation of the image capturedevice between the two images may be measured. Accordingly, the distanceto the object can be estimated.

In accordance with embodiments of the present disclosure, a distance tothe object or objects of interest may be determined (e.g., measured). Asdepicted in at least FIG. 1, embodiments in accordance with the presentdisclosure utilize a mobile device, such as a smartphone 108, to obtaina plurality of images of an object, such as an object of interest 104.Of course, other objects, such as object 128, may be imaged as well. Anestimated distance between an object in smaller and smaller regions oftwo images may be compared in an iterative manner and may result in aphysical distance. As one example, both images may be iterativelysubdivided in a same or similar way and an estimated distance movedbetween the object in the first image and the object in the second imagemay be determined for each subdivision. In this way, relative movementrates are established for arbitrarily small features in the images.Similar to the way the human brain processes images, the embodiments ofthe present disclosure are able to differentiate objects from oneanother at least in part by these relative movement differences. Anoverall average of estimated movement (the center of rotation), as wellas a map of differences between smaller features of the images, isobtained; each of these may be utilized in a distance calculation usinga known difference in an angle of rotation between first and second(subsequent) images.

At least one embodiment in accordance with the present disclosureincludes an image capture device, a storage medium, a processing unit,and a means of monitoring a relative position of the image capturingdevice, such as a global positioning system (“GPS”) or gyrometer.Embodiments of the present disclosure can therefore function on anyhardware that includes these features, such as a smartphone.Accordingly, embodiments of the present disclosure include a means ofrelating these technologies in order to identify a central object frommany different angles. Thus, a three-dimensional virtual model may becreated by: (1) a user identifying a three-dimensional object; (2)capturing a plurality of two-dimensional images of said object insuccession, said plurality of images being captured from differentorientations: (3) recording said plurality of images on a storagemedium; (4) determining the relative change in position of saidplurality of images by comparing two subsequent images; wherein therelative change is determined by a difference in color intensity valuesbetween the pixels of one image and another image; (5) generating aplurality of arrays from the difference determined; and (6) generating acomputer image from said plurality of arrays, wherein said computerimage represents said three-dimensional object.

Embodiments of the present disclosure may therefore utilize existingrecording hardware, positional sensor, processor, and storage medium,all of which may exist within the same unit. Certain embodiments involvea capture thread, which is how a user interacts with a camera with the3D model-capturing software application. For instance, a user maydownload and install an application on a mobile device 108 or equivalenthardware. The user starts the application and uses hardware in themobile device 108 to measure the object of interest 104. As depicted inFIG. 1, the object of interest 112 may be displayed on a display of themobile device 108.

In accordance with embodiments of the present disclosure, an identitymatrix is set up in which all measurements will be referenced andfurther, an empty “first in, first out” (FIFO) stack is created. Thus, acapture thread may include: (1) begin capturing images; (2) checking tosee if the user actor has terminated the scan; (3) capture an imageusing an image capture device (4) measuring the current camera'sorientation; (5) creating a 4×4 transformation matrix referenced fromthe initial identity matrix using recently measured orientations andtranslations by the camera; (6) pushing the current image andtransformation matrix onto a FIFO stack; (7) rotating the camera inplace without moving the camera side-to-side; and (8) repeating steps1-7 wherein a 3D image is produced. Embodiments of the presentdisclosure may use sensors such as gyroscopes, magnetometers, and otherrotation sensors to measure the camera's orientation.

Embodiments of the present disclosure may be executed on a mobile device108 such as a mobile phone that is already equipped with a processingunit capable of carrying out computational operations, a digital camera,a storage medium, and a gyrometer or accelerometer. As depicted in FIG.2, the mobile device 108 may include at least one processor/controller204, one or more memories 208, a communication interface 212 and antenna216, a power source 220, at least one camera 224, input 228, output 232,program storage 236, one or more sensors 240, one or more filters 244,and one or more buses 248. However, in other embodiments, the mobiledevice 108 includes only some of the components 204-248. For example, inone embodiment, the mobile device 108 includes the at least oneprocessor/controller 204, the one or more memories 208, thecommunication interface 212 and antenna 216, the power source 220, theat least one camera 224, the input 228, the output 232, the programstorage 236, the one or more sensors 240, and one or more buses 248. Ingeneral, the mobile device 108 may include any suitable combination ofthe components 204-248. In one application, mobile device 108, is avideo telephony device (e.g., video phones, telepresence devices, acamera-equipped cellular or wireless phone, a mobile collaborationdevice, and a personal tablet, or laptop computer with a camera or webcamera).

The processor/controller 204 may be capable of executing programinstructions. The processor/controller 204 may include anygeneral-purpose programmable processor or controller for executingapplication programming. Alternatively, or in addition, theprocessor/controller 204 may comprise an application specific integratedcircuit (ASIC). The processor/controller 204 generally functions toexecute programming code that implements various functions performed bythe mobile device 108 in accordance with at least some embodiments ofthe present disclosure.

The mobile device 108 may additionally include one or more memories 208.The one or more memories 208 may be used in connection with theexecution of programming instructions by the processor/controller 204,and for the temporary or long-term storage of data and/or programinstructions. For example, the processor/controller 204, in conjunctionwith the one or more memories 208 of the mobile device 108, may operateto execute one or more instructions to capture a plurality of images ofan object of interest 104, process the images, and providethree-dimensional coordinates of said object of interest 104 inaccordance with embodiments of the present disclosure.

The one or more memories 208 of the mobile device 108 may comprisesolid-state memory that is resident, removable and/or remote in nature,such as DRAM and SDRAM. Moreover, the one or more memories 208 maycomprise a plurality of discrete components of different types and/or aplurality of logical partitions. In accordance with still otherembodiments, the one or more memories 208 comprises a non-transitorycomputer-readable storage medium. Such a medium may take many forms,including but not limited to, non-volatile media, volatile media, andtransmission media.

The mobile device 108 may include a user interface allowing a user tointeract with the mobile device 108, to operate the mobile device 108,and/or to interact with a feature, function, and/or application of themobile device 108. For example, a user of the mobile device 108 mayinteract with an application running on the mobile device 108 to recordmultiple images of the object of interest 104. Examples of input 228 mayinclude user input devices such as but not limited to a keypad, a touchscreen, a microphone, and a pointing device. Examples of output 232 mayinclude but are not limited to user output devices such as a display124, which may be a touch screen display, a speaker, and one or morehaptic output devices.

The mobile device 108 may be equipped with a communication interface 212and antenna 216. The communication interface 212 may comprise a GSM,CDMA, FDMA and/or analog cellular telephony transceiver capable ofsupporting voice, multimedia and/or data transfers over a cellularnetwork. Alternatively, or in addition, the communication interface 212may comprise a Wi-Fi, BLUETOOTH™, WiMax, infrared, NFC or other wirelesscommunications link. The communication interface 212 may be associatedwith one or more shared or a dedicated antenna 216. The type of mediumused by the mobile device 108 to communicate with other portableelectronic devices may depend upon the communication application'savailability on the mobile device 108 and/or the availability of thecommunication medium.

The mobile device 108 may include one or more sensor(s) 240, such as oneor more accelerometers and/or one or more gyroscopes. An accelerometermay comprise any device that detects acceleration forces, usually in alinear direction along one or more axes. In general, an accelerometermay have the ability to gauge an orientation of the communication devicerelative to the Earth's surface. For example, most accelerometersinclude the ability to detect acceleration forces exerted with respectto one or more axes, such as the X-axis, Y-axis, and Z-axis. Thus, theaccelerometer may actually comprise one or more individualaccelerometers that measure the acceleration forces with respect to eachaxis. In general, when referring to an accelerometer or the dataprovided by the accelerometer, the accelerometer is assumed herein tohave the capability to provide data or information regardingacceleration forces exerted on at least one of the X-axis, Y-axis, andZ-axis. Thus, the accelerometer can detect an angle at which the mobiledevice 108 is being held, measure movements such as rotation, motiongestures, shaking, and flicking of the mobile device 108 by sensing theacceleration forces exerted on each axis. An example of an accelerometermay include, but is not limited to, a Microelectromechanical System(MEMS) accelerometer, such as a STMicroelectronics STM331DLH.

As previously mentioned, the sensor(s) 240 may further include one ormore gyroscopes. A gyroscope generally measures the rate of rotation, orthe angular acceleration, around an axis. For example, a three-axisgyroscope may measure the angular acceleration around each of theX-axis, Y-axis, and Z-axis, enabling the precise calculation of yaw,pitch, and roll. An example of a gyroscope may include, but is notlimited to, a Microelectromechanical System (MEMS) gyroscope, such asthe STMicroelectronics L3G4200D. Although the sensor may include otherinstruments, such as a magnetometer, sensor(s) 240 may comprise athree-axis accelerometer and a three-axis gyroscope such that the mobiledevice 108 is capable of calculating how far, how fast, and in whatdirection the mobile device 108 has moved in a space.

Data from the one or more sensors 240 may be filtered utilizing anoptional filter 244. The filter 244 may filter, or otherwise smooth, thedata received from the one or more sensors 240 such that variations doin part to sensor drift, temperature, biases, and accuracy can besufficiently accounted for. In one instance, the one or more sensors 240may employ an Extended Kalman Filter to account for such biases and tomaintain accuracy of the received measurements from a gyroscope forexample.

The camera 224 may comprise any camera generally included in a mobiledevice 108 such as a smartphone or the like. In general, the cameraresolution may be 1024×768 pixels or more and may be utilized to obtainone or more images, such as a video, of the object of interest 104.

In certain embodiments, the camera 224 is used to record video of anobject. A user “sweeps” the phone back and forth across the object,allowing the camera 224 to record the object from many angles. Whileimages are recorded, the position or inertial detection (such as agyrometer, accelerometer, magnetometer, or other means of relativeposition identification) records relative or absolute positionalchanges. Relative changes in position are recorded for subsequent stillframes in a video sequence. These changes are hereafter referred to as“θ” (Greek letter theta). Each still frame in the video sequencetherefore has a θ value associated with it, indicating how far in real,physical space the recording device has moved since capturing theprevious image. θ is therefore a measurement of the angle formed betweentwo subsequent images of the same, central object.

Embodiments of the present disclosure also include identifying relativechanges to virtual distances referred to as “d” 120 as shown in FIG. 1.This aspect of at least one embodiment uses a process referred to hereinas pixel mapping. The distance between images acquired is indicated byidentifying relative changes to virtual distances. Different elements ofthe image being recorded change between the two images, such as theposition of the cup handle, and the corner of the table on which the cupsits, while the table in the foreground of the images changes verylittle. These changes will be identified in the pixel mapping processand used to estimated 120 and assign relative, three-dimensionalpositions.

In accordance with embodiments of the present disclosure, pixel mappingcompares two subsequent images and uses changes in pixel color values toestimate a distance between them. In this process, pixel values are readinto an array for two subsequent images, and these values are stored inmemory 208. As each image is captured in succession, it is added to aFIFO queue. The red, green, and blue values for each pixel from each ofthe two images are then compared by subtraction at each position. Thisprocess generates a new array, in which the matrix of difference valuesat each position is stored. This new array is summed, and the squareroot of the sum is reported as a cost function as described in FIG. 8.As illustrated in at least FIG. 3A, a first image A is obtained of anobject of interest 104. Of course, the images presented in FIGS. 3A-Bare for illustrative purposes and such resolution is in no way intendedto limit the scope or operation of the embodiments described herein. Asillustrated in FIG. 3A, the images' pixel values between image A andimage B are subtracted and an array illustrated in image C is obtained.Images A, B, and C, of FIG. 3A do not show pixel values for all pixelsin order to avoid unnecessarily obscuring the present disclosure. Aspreviously discussed, the pixel values may comprise Red, Green, Blue(RGB) values. In some embodiments, a combination of RGB values may besubtracted from one another; in other embodiments, separate red imagesmay be subtracted from one another, separate blue images may besubtracted from one another, and separate green images may be subtractedfrom one another. Of course other colorspaces, such as but not limitedto YCbCr, YPbPr, and cmyk may be utilized.

The cost function value is used as input for a simplex optimizer, whichrequires a single value. The simplex optimizer may be an algorithm thatminimizes the cost function, and the pixel mapping process is compatiblewith any available simplex optimizer. In certain embodiments, thesimplex optimizer shifts one of the two images relative to the other bya number of pixels, and recomputes the cost function. Once asufficiently low enough cost function value is achieved such thatsubsequent trials are unable to produce a lower value, according to auser-defined threshold, the optimizer terminates. The optimizer thenreports the lowest value obtained, along with the change in X (ΔX) andchange in Y (ΔY) pixel shift values used to obtain that value. Thisvalue represents the distance between the center-most pixel in each ofthe two images.

Once d 120 has been calculated in this way, it can be used in thefollowing equation along with the measured 0 value in accordance withEquation 1.

$\begin{matrix}{r = \left( \frac{d}{\theta} \right)} & {{Equation}\mspace{14mu} 1}\end{matrix}$

As illustrated in Equation 1, the r value is used to establish a centralreference point, such as (0,0, r). Once this frame of reference isestablished, the two images are then subdivided into quadrants asillustrated in FIG. 3B and d 120 is again calculated between each of thefour quadrants as shown in FIG. 3B, again using optimization. In thisembodiment, calculation of d 120 is carried out concomitantly duringimage acquisition; accordingly, image acquisition and feature isolationand/or movement may be determined at a same time. By successivelysubdividing the image into smaller regions of analysis and evaluatinghow far each element has moved, the valuated value therefore estimatesthe way in which the frame of reference is moving in physical space.This process of subdividing the images and recalculating their distancesis repeated until no more similarities are found. Because some contentis lost and other content is gained as the camera moves from one side toanother, not all regions of the image will have distance valuesassociated with them, because they cannot be mapped. However, as theprocess continues through smaller and smaller iterations, individualfeatures of an image will be identified and their distances calculated,as shown in FIG. 3B. This entire process is repeated for each subsequentimage.

In accordance with embodiments of the present disclosure, at least oneembodiment makes use of an analysis of variance. By evaluating whichelements have moved more quickly between frames, a simple statisticaltest of variance, such as least squares regression, can further informthe system of the relative distance of different objects in frame fromthe observer. This is based on the observation that objects near to theobserver appear to move more quickly than objects far from the observerwhen the observer moves laterally to them. Once pixel mapping has beencompleted and different features of the images have been identified, astatistical test such as a regression analysis can be used to determinehow those features are moving relative to one another, and thereforeprovide added support for the estimate of distance from the observer.

As depicted in FIG. 4, and for the purposes of this disclosure, anobject being scanned may be defined as being at the origin of anarbitrary grid system. A vector describing a relative position of thecamera within the mobile device with respect to the object being scannedis the physical distance between two successive images recorded by thecamera and is referred to as d in FIG. 1 and FIG. 4, and the anglebetween these two images is θ. The coordinate system includes theobject(s) of interest at its origin. As the user scans the object, thedistance between any two successive images is represented by d, and theangle between said images is represented by θ. The distance from theobject(s) of interest to the user is the vector, and the half angle isrepresented by forming a right triangle. It may be assumed that θapproaches 0 when d is very small.

The coordinates of r are therefore defined according to Equation 2.

$\begin{matrix}{\overset{\rightarrow}{r} = \left( \frac{x}{y} \right)} & {{Equation}\mspace{14mu} 2}\end{matrix}$

The magnitude of r is defined according to Equation 3.

r=√{square root over (x ² +y ²)}  Equation 3

The half angle between two successive images forms a right angle and isdefined as ρ. This secondary vector is similarly defined according toEquation 4.

$\begin{matrix}{\overset{\rightarrow}{\rho} = \left( \frac{x}{y} \right)} & {{Equation}\mspace{14mu} 4}\end{matrix}$

The magnitude of ρ is defined according to Equation 5.

ρ=√{square root over (x ² +y ²)}  Equation 5

ρ can be determined empirically from the rearranged trigonometricidentity according to Equation 6.

$\begin{matrix}{\rho = \frac{d}{2{\tan\left( \frac{\theta}{2} \right)}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

Although r≠ρ, they are similar vectors when 0 is small, and approach thesame value as θ approaches 0. By using live imaging, embodiments inaccordance with the present disclosure make use of very small values of0, and therefore assume r≈ρ.

To further simplify the calculations that must be performed by theprocessor, in accordance with at least some embodiments of the presentdisclosure, a Taylor Series expansion can remove all higher orderedpolynomial terms ad provided in Equation 7.

$\begin{matrix}{{\sin\;\theta} = {{\theta - \left( \frac{\theta^{3}}{3!} \right) + \left( \frac{\theta^{5}}{5!} \right) - {\ldots\mspace{14mu}\cos\;\theta}} = {{1 - \theta - \left( \frac{\theta^{2}}{3} \right) + \left( \frac{\theta^{4}}{4!} \right) - {\ldots\mspace{14mu}{\tan(\theta)}}} = \left( \frac{\sin\;\theta}{\cos\;\theta} \right)}}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

Because of this, the following assumptions produce linear—and thereforemore simplified—terms: sin θ≈θ cos θ≈1 tan θ≈θ. Using theseapproximations, the following assumption is produced:

$\begin{matrix}{r = \left( \frac{d}{\theta} \right)} & {{Equation}\mspace{14mu} 8}\end{matrix}$

In certain embodiments, d and θ are measured empirically as previouslydescribed, and r is computationally derived. A simplified calculationcan be executed far more rapidly than the otherwise necessary non-linearcalculations that would be required. By assuming very small lateraldistances around the object, certain embodiments simplify themathematical operations enough such that they can be performed quicklyby most modern small computational units, such as smartphones.Accordingly, virtual coordinates can be generated for any real object inphysical space. Certain embodiments can generate a virtual,three-dimensional model at a level of resolution limited only by theresolution of the camera and the distance of the camera to the objectbeing scanned. These coordinates can be used to generate a virtual modelin any format of the object being scanned.

In accordance with embodiments of the present disclosure, the creationof a 3D image can be performed in accordance with the followingnon-limiting steps:

1. Take the first picture;1.1. Add picture to a (first in, first out) FIFO queue;2. Take the next picture;2.1. Add picture to a (first in, first out) FIFO queue;3. While still taking pictures (at some defined frequencyframes/second), loop back to step 2;4. In a separate thread of execution (while pictures are being taken insteps 1-3), do the following:

-   -   4.1. Pop the next picture off the queue (this is picture 1);    -   4.1.1. Read the images pixel array and save;    -   4.2. Read the next picture off the queue (this is picture 2);    -   4.2.1. Read the images pixel array and save;    -   4.3. Define a cost function (the function we want to minimize);    -   4.3.1. Using the pixel arrays of picture 1 and 2;    -   4.3.2. And given a delta-x and delta-y pixel shift amount;    -   4.3.2.1. Shift picture 1 by this amount;    -   4.3.3. Loop over the arrays subtracting the value of each pixel        and save the differences in a new array;    -   4.3.4. Compute the magnitude (normalize) the new array by        treating it as a vector;    -   4.3.5. Return this magnitude (imagine, the more the pictures        look alike, the lower this number is);        4.4. Utilize an optimizer using the cost function in step 4.3.        This can be any optimizer that will find an optimal, or at least        one, solution (delta-x and delta-y) to shift picture 1 to        picture 2. As one non-limiting example, a nonlinear programming        simplex optimizer may be utilized;        4.5. The optimizer will continue to search for the best        solution. This step is similar to the example in the last        section where it is trying to move the film of picture 1 on top        of picture 2 until the pictures map to each other as best as        they can;        4.6. Once the optimal solution is found, the distance between        the center most pixel on both pictures is known;        4.7. Compute the three-dimensional point, similar to (0, 0, r)        where r is computed from Equation 8. This three-dimensional        point is referenced in the user's camera frame defined in FIG.        4;        4.8. Subdivide the two pictures into quarters by dividing their        arrays from steps 4.1.1 and 4.2.1;        4.9. Loop over each new sub-divided array, and pass each new        pixel array to step 4.4 then execute steps 4.5 and 4.6.        4.9.1. The solution of the center most pixel in the sub-divided        picture is determined;        4.9.2. Using spherical coordinate transformations from Cartesian        space, the new three-dimensional points may be rotated from the        center of the pictures to where the pixels are located.        Accordingly, there are four additional measurements.        4.10. Continue to sub-divide each picture into smaller quarters        by looping over step 4.9 until the shifting of the pixels is not        returning any optimal results. Steps 4.9 and 4.10 find distances        between the different features of the object;        4.11. Loop back to step 4.1 until no more pictures remain on the        queue from steps 1 to 3;        5. When complete, all the three-dimensional points computed are        stored for further post-processing;        5.1. Using all the points measured, conduct a least-squares        algorithm (LSQ) to weed out any outlying points that do not fit        within a data-driven sigma value and smooth out the points        (using interpolation techniques, like cubic-spine algorithm);        save all the points as a three-dimensional model that can be        viewed/rendered using software on a computer or printer.

By truncating the Taylor Series approximation at the second power, it isassumed to carry error on the order of three. A conservative estimate istherefore that the assumption of a small θ is reasonably safe up toapproximately 10 degrees of difference between successive images.Because most modern cameras use a minimum of 24 frames per second, anysweeping motion at a reasonable pace will produce images that are wellbelow 10 degrees of separation. Embodiments in accordance with thepresent disclosure are therefore not limited by these assumptions in anyreasonable usage. Embodiments in accordance with the present disclosurecan therefore objectively differentiate any object from any absoluteangle, as long as successive images are recorded in close proximity toone another, such as through the use of video recording.

Certain embodiments use a streamlined method in a standard mobile phoneequipped with a camera and gyrometer. In at least one embodiment, a userinitiates the imaging software, which invokes the imaging hardware, forexample, a camera. The user then aims the camera toward the object orobjects of interest and initiates recording using an on-screen button.

Once in recording mode, the user then moves the phone in such a way asto record the object or objects of interest from many angles. Althoughthere is no minimum number of images acquired at different angles thatare required, fidelity is improved with a greater number of angles. Inthis embodiment, the software is also better able to distinguish theobject or objects of interest from the background through the use of agreater number of images acquired at different angles by way of ananalysis of variance, as described above.

Once the user is satisfied with the length of recording in thisembodiment, the user then stops the recording (while still recording theobject or objects of interest), using an on-screen button. Imagecalculation runs concomitantly during image acquisition, and does notrequire separate user initiation. In certain embodiments, this processcontinues beyond the cessation of image acquisition, if necessary, foras long as is required to process all of the acquired images.

Once initial processing is complete, post hoc processing isautomatically engaged. Alternative embodiments make this continuationstep optional, allowing the user to instead carry out any post hocanalysis at a later time and on different hardware, if desired. Theinitial analysis produces data related to the three-dimensional locationof the object or objects of interest, and can be consideredindependently of other analyses. Any subsequent analyses carriedfunction to further improve fidelity of the virtual model. The modularnature of this algorithm is such that post hoc analyses are not requiredfor the algorithm to function.

In certain embodiments, post hoc analysis is carried out and results ina table of coordinates of vector points, and includes data on theirconnectivity to other vector points. Data is formatted in standard 3Dfile formats. This data can be saved or exported to alternative fileformats for importation into other software programs.

In certain embodiments, the user walks around the object(s) of interestwith a mobile phone while a camera monitors the object(s) and apositional sensor within the phone tracks relative changes to position.This can be very useful in rapid prototyping and 3D printing. Forexample, if a user breaks the handle off of their coffee mug. Theembodiment could be used to scan both the mug and the handle. Thedigital representations of these elements could be adjoined as one andsent as instructions to a 3D printer.

In other embodiments, movement is not restricted to a revolving motion,and may instead comprise a sweeping motion, in which a user passes thecamera back and forth across one area of the object or objects.Movements involved in the recording process are unrestricted, and maycomprise a user standing in one position, while moving a mobile phoneback and forth in a sweeping motion. Such embodiments may involveinstances in which it is not feasible or necessary to revolve an entireobject. For example, if a user wishes to replace a door and wants torecord the dimensions of a door-frame, the user is not required to walkaround the door-frame. In this example, the user stands in one positionand passes the camera back and forth across the field of view thatencompasses the door frame. In instances such as these in which crispdetail is not necessary for determining dimensions, post hoc analysismay be entirely bypassed.

Other embodiments may not be implemented on a mobile device 108. Forexample, an airborne drone equipped with a camera and a positionaltracking system (such as a GPS system, for example) might be used toinvestigate remote regions difficult or dangerous for human presence. Inthese embodiments, the drone may be equipped with on-board processingand storage media. Alternatively, the drone may not be equipped withthese elements, and may instead transmit the data to an alternativelocation for storage and processing, or be equipped with both, and becapable of switching between formats.

Similarly, satellites investigating celestial bodies may alsoincorporate the disclosed method. For example, a satellite investigatinga large crater on a moon might generate a three-dimensional map of thecrater, revealing detail too difficult to discern in two-dimensionalimages. In these embodiments, additional post hoc analysis might befurther incorporated. For example, one embodiment might integrateparallax or other data to generate a three-dimensional map of a distantregion of the Universe instead of a near-Earth celestial body. The unitsin this embodiment could be in light years, and the three-dimensionalmap could be used to track celestial movements. In any of theseembodiments, a critical advantage of this disclosure exists in thatsatellites need not be equipped with additional hardware. This featureof the method makes it possible for existing satellites previouslylaunched to begin generating three-dimensional maps immediately withoutreturning to Earth for hardware modifications. The method makes use ofexisting hardware, including an image acquisition feature, a celestialcoordinate system, and an ability to transmit data, each already in use.

Referring now to FIG. 5, a method 500 of creating a three-dimensionalvirtual image using a single camera, such as camera 224 of a mobiledevice 108 will be discussed in accordance with embodiments of thepresent disclosure. Method 500 is in embodiments, performed by a device,such as a mobile device 108. More specifically, one or more hardware andsoftware components may be involved in performing method 500. The method500 may be executed as a set of computer-executable instructionsexecuted by a computer system and encoded or stored on acomputer-readable medium. Method 500 may be executed utilizing theprocessor/controller 204 and/or the memory 208 of the mobile device 108.Hereinafter, the method 500 shall be explained with reference tosystems, components, modules, software, etc. described with FIGS. 1-4.

Method 500 may continuously flow in a loop, flow according to a timedevent, or flow according to a change in an operating or statusparameter. Method 500 is initiated at step S504 where a mobile device108 may start one or more image recordation operations in response to auser input, for example at input 228. At step S508, the mobilecommunication device 108 may obtain a plurality of images using thecamera 224; the image may be stored in a FIFO queue for example, withinthe one or more memories 208 for instance. Such plurality of images isgenerally sequentially acquired and may include a timestamp. As oneexample, T₁ and T₂ of FIG. 1 may be sequentially acquired, wherein T₂ isacquired after T₁.

At step S512, two images may be selected; the two images selected may beadjacent in time; that is the second image may have been obtained rightafter the first image. For example, T₂ and T₁ of FIG. 1 may be selected.In some embodiments, the two selected images may not be adjacent to oneanother in time; instead, one or more images may exist between the twoselected images. In accordance with at least some embodiments of thepresent invention, the two images may be selected based on a timestamp.

Flow may continue at step S516, where one or more methods, steps, and/orprocesses may be utilized to determine a distance between the centermost pixels of the two selected images. As previously described, adifference array between the first and second image may be obtained andan optimizer utilizing one or more cost functions may find the lowestcost solution such that an X,Y shift amount is obtained. Based on theX,Y shift amount, a distance between the center most pixel of bothpictures can be obtained. Accordingly, a three-dimensional point may becomputed using Equation 9 for example.

Flow may continue to step S520 where features of the object of interest104 may be obtained and distances between said features may bedetermined. In accordance with at least one embodiment of the presentdisclosure, each of the two pictures may be subdivided. As one example,each of the two pictures may be subdivided into quarters where adistance between center most pixels of the subdivided images from thefirst and second images may be obtained. As another example, each of thetwo pictures may be subdivided into halves where a distance betweencenter most pixels of the subdivided images from the first and secondimages may be obtained. Similar to step S516, a difference array betweenthe subdivided images of the first and second images may be obtained andan optimizer utilizing one or more cost functions may find the lowestcost solution such that an X,Y shift amount is obtained. Based on theX,Y shift amount, a distance between the center most pixel of bothsubdivided images of the first and second images can be obtained. StepS520 may loop in an iterative manner such that each smaller image isfurther subdivided into a predetermined number of images and distancesbetween each of the center most pixels may be obtained. At step S524,three-dimensional coordinates for each of the obtained centermost pixelsmay be determined. Further, outlying points may be removed using aleast-squares algorithm and points may be smoothed. At step S528, ifthere are additional pictures in the queue, method 500 may proceed backto step S512 where steps S512, S516, S520, and S524 are repeated. Insome embodiments, an output display 124 of the mobile device 108 maydisplay the newly determined three-dimensional points in a window on thescreen as will be described later. At step S528, if no additionalpictures remain in the queue, method 500 flows to step S532 where athree-dimensional model of the object of interest 104 is generated. Thethree-dimensional model of the object of interest 124 may be assembledutilizing the determined three-dimensional points of the variousfeatures; such points may be supplied to a shader to render thethree-dimensional model. Method 500 may end at step S536.

Referring now to FIG. 6, a method 600 for performing image capture usinga single camera, such as camera 224 of a mobile device 108 will bediscussed in accordance with embodiments of the present disclosure.Method 600 is in embodiments, performed by a device, such as a mobiledevice 108. More specifically, one or more hardware and softwarecomponents may be involved in performing method 600. The method 600 maybe executed as a set of computer-executable instructions executed by acomputer system and encoded or stored on a computer-readable medium.Method 600 may be executed utilizing the processor/controller 204 and/orthe memory 208 of the mobile device 108. Hereinafter, the method 600shall be explained with reference to systems, components, modules,software, etc. described with FIGS. 1-5.

Method 600 may continuously flow in a loop, flow according to a timedevent, or flow according to a change in an operating or statusparameter. Method 600 is initiated at step S604 where a mobile device108 may start one or more image recordation operations in response to auser input, for example at input 228. At step S608, the mobilecommunication device 108 may obtain a first image, such as image A. Atstep S612, the acquired image A may be added to a queue such as a FIFOqueue. At step S616, another image, image B, may be obtained and may beadded to the same FIFO queue at step S620. The FIFO queue may residewithin the one or more memories 208. The process of obtaining images andadding the images to one or more queues may continue at step S624 untilsome threshold, such as recording time, file size, free space, and/orscanning enabled button indicates that the image acquisition processshould stop. At step S632, method 600 may end. Method 600 may generallybe utilized at step S508 of FIG. 5.

Referring now to FIG. 7, methods 700A and 700B for determiningthree-dimensional reference points using a mobile device 108 will bediscussed in accordance with embodiments of the present disclosure.Methods 700A and 700B are in embodiments, performed by a device, such asa mobile device 108. More specifically, one or more hardware andsoftware components may be involved in performing methods 700A and 700B.The methods 700A and 700B may be executed as a set ofcomputer-executable instructions executed by a computer system andencoded or stored on a computer-readable medium. Methods 700A and 700 Bmay be executed utilizing the processor/controller 204 and/or the memory208 of the mobile device 108. Hereinafter, the methods 700A and 700Bshall be explained with reference to systems, components, modules,software, etc. described with FIGS. 1-6.

Method 700A may continuously flow in a loop, flow according to a timedevent, or flow according to a change in an operating or statusparameter. Method 700A is initiated at step S704 where a mobile device108 may initiate the determination of a three-dimensional referencepoint. At step S708, a first image, such as image A, may be pulled fromthe queue and a pixel array may be determined and stored in a new firstarray at step S712. At step S716, a second image, such as image B, maybe pulled from the queue and a pixel array may be determined and storedin a new second array at step S720. At step S724, a difference arraybetween the first and second image may be obtained and an optimizerutilizing one or more cost functions may find the lowest cost solutionsuch that an X,Y shift amount is obtained. Based on the X,Y shiftamount, a distance between the center most pixel of both pictures can beobtained at step S728. Accordingly, a three-dimensional reference pointmay be computed using Equation 9 for example. Method 700A may end atstep S732.

Method 700B may continuously flow in a loop, flow according to a timedevent, or flow according to a change in an operating or statusparameter. Method 700B is initiated at step S736 where a mobile device108 may initiate the determination of a three-dimensional point forfeatures pertaining to an object of interest 104. At step S740, thefirst image and the second image may be subdivided. As previouslydiscussed, the images may be subdivided into any number of smallerimages 304A and 304B for example. As one example, the images may besubdivided into quadrants.

At step S744, a difference array between the subdivided images of thefirst and second images may be obtained (308 for example) and anoptimizer utilizing one or more cost functions may find the lowest costsolution such that an X,Y shift amount is obtained. Based on the X,Yshift amount, a distance between the center most pixel of bothsubdivided images can be obtained at step S748. At step S752, each ofthe subdivided images may be further subdivided again. That is, method700B may iterate through steps S740 to S748 further subdividing eachsubdivided image until a specific threshold amount is achieved. That is,method 700B may continue to subdivide images and repeat steps S740 toS748 until not additional optimized features are obtained, apredetermined number of subdivides is obtained, and/or a minimize costdetermined by a simplexer is within a predetermined threshold range.Accordingly, at step S756, spherical coordinate transformations fromCartesian space may be performed and the new three-dimensional pointsmay be rotated from the center of the pictures using the previouslydetermined three-dimensional reference point determined at step S728.Method 700B may end at step S770 where such points are provided to stepS536 of FIG. 5.

Referring now to FIG. 8, a method 800 implementing a cost function willbe discussed in accordance with embodiments of the present disclosure.Method 800 is in embodiments, performed by a device, such as a mobiledevice 108. More specifically, one or more hardware and softwarecomponents may be involved in performing method 800. The method 800 maybe executed as a set of computer-executable instructions executed by acomputer system and encoded or stored on a computer-readable medium.Method 800 may be executed utilizing the processor/controller 204 and/orthe memory 208 of the mobile device 108. Hereinafter, the method 800shall be explained with reference to systems, components, modules,software, etc. described with FIGS. 1-7.

Method 800 may continuously flow in a loop, flow according to a timedevent, or flow according to a change in an operating or statusparameter. Method 800 is initiated at step S804 where a mobile device108 may perform a cost function with respect to first and second images,for example image A and image B. At step S808, a first array of pixelvalues, such as RGB, may be loaded from a first image (image A) into amemory location, such as within the one or more memories 208. At stepS812, a second array of pixel values, such as RGB, may be loaded from asecond image (image B) into a memory location, such as within the one ormore memories 208. At step S816, a difference array including differencevalues may be generated; the difference values corresponding to adifference between pixel values in the first image and pixel values inthe second image. For example, the R values of the second image may besubtracted from the first image. At step S820, the difference values maybe stored in a new third array. At step S824, all values in the thirdarray may be summed and a square root of the result may be provided atstep S828. In some instances, method 800 may end at step S832. Inaccordance with other embodiments of the present disclosure, method 800may continue to step S840 where an optimizer may determine whether ornot the square root of the result has been minimized. In some instances,the images may be shifted by one or more pixels in a first and/or seconddirection at step S836 and steps S816 to S828 may be performed again. Insome instances, a simplexer as previously described may minimize thecost function. Once the cost function has been sufficiently minimized,such as within a predetermined range of a threshold, method 800 may endat step S832.

FIG. 9 generally depicts a second system for creating athree-dimensional virtual image in accordance with at least someembodiments of the present disclosure. In accordance with at least someembodiments of the present disclosure, part of the method for creating athree-dimensional virtual image may be performed on a device other thanmobile device 108. That is, the method may be distributed across atnetwork 912, where a first set of servers 908 may perform part of themethod and/or another device, such as mobile device 904 may performanother part of the method. For example, mobile device 108 may obtain aplurality of images; mobile device 108 may transfer some or all of theplurality of images across the network 912 to one or more server devices908. The server device 908 may utilize the plurality of images to createa three-dimensional model in accordance with one or more previouslydescribed methods. The three-dimensional model may then be sent back tothe mobile device 108. Alternatively, or in addition, another device,such as device 904 may perform part of the three-dimensional modeldevelopment. For instance, a plurality of images may be sent to theserver devices 908 and a plurality of the images may be sent to theother device 904. Both server devices 908 and mobile device 904 maydetermine three-dimensional coordinate points and such points may becombined and provided to mobile device 108. Mobile device 108, oranother device communicatively coupled to mobile device 108, may createthe three-dimensional model of the object of interest 104 using one ormore shaders.

The server device 908 and/or the other device 904 may generally includethe same elements as those found and described within the mobile device108 in FIG. 2. Moreover, the network 912 may include one or morenetworks capable of supporting communications between nodes of thesystem 900. Examples of communication networks 912 include the Internetor any wide area network (WAN), local area network (LAN), or networks invarious combinations. Other examples of communication networks 912 inaccordance with embodiments of the present disclosure include wirelessnetworks, cable networks, satellite networks, and digital subscriberline (DSL) networks. In addition, different communication networks 912may share a common physical infrastructure, for at least some portion ofthe physical network. For instance, a mobile device 108 may beinterconnected to a communication network 912 comprising the Internetand to a separate communication network between the device 904 andmobile device 108.

FIG. 10 generally depicts a display 124 of a mobile device 108 inaccordance with at least some embodiments of the present disclosure. Aspreviously described, the mobile device 108 may display an image 112 ofthe object of interest 104 on the display 124. The display 124 may alsoinclude an information box 1004 including a status bar 1012 and apartial and/or complete rendering of the three-dimensional model 1008 ofthe object of interest 104. The status bar 1012 generally provides anindication as to a number of acquired three-dimensional points verses anestimated number of three-dimensional points needed for a completemodel. Alternatively, or in addition, and as further depicted in FIG.10, a number of acquired three-dimensional points may be displayed. Forinstance, the information box 1004 indicates that 1560three-dimensionally acquired.

The information box 1004 further includes a partial and/or completerendering of the three-dimensional model 1008 of the object of interest104. As previously discussed, the complete rendering of thethree-dimensional model 1008 of the object of interest 104 may begenerally displayed over time as three-dimensional points of athree-dimensional model are acquired and determined. Accordingly, asdepicted in FIG. 10, because not all three-dimensional points of theobject of interest 104 have been determined, the rendering of the objectof interest 1008 within the information box 1004 is incomplete. Suchincomplete drawing further corresponds to the status bar 1012 indicatingthat additional three-dimensional points are needed. FIG. 10additionally depicts a bounding box 116.

As previously described, a least-squares algorithm (LSQ) may be utilizedto weed out any outlying points that do not fit within a specificlocation. The bounding box 116 may define the specific location of whichoutlying points not within the bounding box 116 may be removed.Accordingly, the need to perform a least-squares algorithm may bedramatically reduced.

In accordance with embodiments of the present disclosure, it may beunnecessary to subdivide the image to acquire three-dimensional pointsof features belonging to the object of interest 104. Instead,three-dimensional points, and thus coordinates, may be acquiredutilizing only the center of the bounding box 116. That is, as a user isobtaining images of the object of interest 104, method 700A may becontinuously implemented such that the center of the first and secondimages corresponds directly to the center of the bounding box 116.Accordingly, over time, only those three-dimensional pointscorresponding to the center of the bounding box 116 may be determined asthe user sweeps the mobile device 108 back and forth to image the objectof interest.

FIG. 11 depicts an additional method of obtaining three-dimensionalpoints in accordance with at least one embodiment of the presentdisclosure. That is, instead of continually subdividing images toaccount for features of an object of interest 104, the rate of movementof features within an image of an object of interest 104 may determinewhat portions of the image are subdivided and/or further how many timessuch image is subdivided. Alternatively, or in addition, the rate ofmovement of features within an image of an object of interest 104 maydetermine optimization thresholds for a particular area of the image.Accordingly, if there is great change between one portion of an imageand another portion of a previous image, a threshold utilized via thecost function and/or simplexer may be lower or tighter than a portion ofan image that has less movement. As depicted in image A of FIG. 11, if ageneral movement of the entire image is to be considered, the ability tofine tune movement based thresholds does not exist when compared toimages B, C, and D of FIG. 11. That is, the more that movement can belocalized to a portion of an image, the more accurate results based onmotion will be.

Alternatively, or in addition, rather than continually subdividing theimage, the image may already be subdivided as depicted in FIG. 11.Accordingly, each region localized and impacted by motion, may bematched independently to a previous image or frame, improving anaccuracy of a resulting three-dimensional point. As previouslydiscussed, the more that movement can be localized to a portion of animage, the more accurate results based on motion will be and thus, themore accurate three-dimensional points will be.

The exemplary systems and methods of this disclosure have been describedin relation to a system and method for generating a virtualthree-dimensional model using a single camera. However, to avoidunnecessarily obscuring the present disclosure, the precedingdescription omits a number of known structures and devices. Thisomission is not to be construed as a limitation of the scopes of theclaims. Specific details are set forth to provide an understanding ofthe present disclosure. It should however be appreciated that thepresent disclosure may be practiced in a variety of ways beyond thespecific details set forth herein.

The embodiments presented herein provide advantages. For example, asingle camera utilized in conjunction with a mobile device may be usedto generate a virtual three-dimensional model of an object of interest.

Furthermore, while the exemplary aspects, embodiments, and/orconfigurations illustrated herein show the various components of thesystem collocated, certain components of the system can be locatedremotely, at distant portions of a distributed network, such as a LANand/or the Internet, or within a dedicated system. Thus, it should beappreciated, that the components of the system can be combined into oneor more devices, such as a mobile device, or collocated on a particularnode of a distributed network, such as an analog and/or digitaltelecommunications network, a packet-switched network, or acircuit-switched network. It will be appreciated from the precedingdescription, and for reasons of computational efficiency, that thecomponents of the system can be arranged at any location within adistributed network of components without affecting the operation of thesystem. For example, the various components can be located in a switchsuch as a PBX and media server, gateway, in one or more communicationsdevices, at one or more users' premises, or some combination thereof.Similarly, one or more functional portions of the system could bedistributed between a telecommunications device(s) and an associatedcomputing device.

Furthermore, it should be appreciated that the various links connectingthe elements can be wired or wireless links, or any combination thereof,or any other known or later developed element(s) that is capable ofsupplying and/or communicating data to and from the connected elements.These wired or wireless links can also be secure links and may becapable of communicating encrypted information. Transmission media usedas links, for example, can be any suitable carrier for electricalsignals, including coaxial cables, copper wire and fiber optics, and maytake the form of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Also, while the flowcharts have been discussed and illustrated inrelation to a particular sequence of events, it should be appreciatedthat changes, additions, and omissions to this sequence can occurwithout materially affecting the operation of the disclosed embodiments,configuration, and aspects.

A number of variations and modifications of the disclosure can be used.It would be possible to provide for some features of the disclosurewithout providing others.

In yet another embodiment, the systems and methods of this disclosurecan be implemented in conjunction with a special purpose computer, aprogrammed microprocessor or microcontroller and peripheral integratedcircuit element(s), an ASIC or other integrated circuit, a digitalsignal processor, a hard-wired electronic or logic circuit such asdiscrete element circuit, a programmable logic device or gate array suchas PLD, PLA, FPGA, PAL, special purpose computer, any comparable means,or the like. In general, any device(s) or means capable of implementingthe methodology illustrated herein can be used to implement the variousaspects of this disclosure. Exemplary hardware that can be used for thedisclosed embodiments, configurations and aspects includes computers,handheld devices, telephones (e.g., cellular, Internet enabled, digital,analog, hybrids, and others), and other hardware known in the art. Someof these devices include processors (e.g., a single or multiplemicroprocessors), memory, nonvolatile storage, input devices, and outputdevices. Furthermore, alternative software implementations including,but not limited to, distributed processing or component/objectdistributed processing, parallel processing, or virtual machineprocessing can also be constructed to implement the methods describedherein.

In yet another embodiment, the disclosed methods may be readilyimplemented in conjunction with software using object or object-orientedsoftware development environments that provide portable source code thatcan be used on a variety of computer or workstation platforms.Alternatively, the disclosed system may be implemented partially orfully in hardware using standard logic circuits or VLSI design. Whethersoftware or hardware is used to implement the systems in accordance withthis disclosure is dependent on the speed and/or efficiency requirementsof the system, the particular function, and the particular software orhardware systems or microprocessor or microcomputer systems beingutilized.

In yet another embodiment, the disclosed methods may be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory, a special purpose computer, a microprocessor, orthe like. In these instances, the systems and methods of this disclosurecan be implemented as program embedded on personal computer such as anapplet, JAVA® or CGI script, as a resource residing on a server orcomputer workstation, as a routine embedded in a dedicated measurementsystem, system component, or the like. The system can also beimplemented by physically incorporating the system and/or method into asoftware and/or hardware system.

Although the present disclosure describes components and functionsimplemented in the aspects, embodiments, and/or configurations withreference to particular standards and protocols, the aspects,embodiments, and/or configurations are not limited to such standards andprotocols. Other similar standards and protocols not mentioned hereinare in existence and are considered to be included in the presentdisclosure. Moreover, the standards and protocols mentioned herein andother similar standards and protocols not mentioned herein areperiodically superseded by faster or more effective equivalents havingessentially the same functions. Such replacement standards and protocolshaving the same functions are considered equivalents included in thepresent disclosure.

The present disclosure, in various aspects, embodiments, and/orconfigurations, includes components, methods, processes, systems and/orapparatus substantially as depicted and described herein, includingvarious aspects, embodiments, configurations embodiments,subcombinations, and/or subsets thereof. Those of skill in the art willunderstand how to make and use the disclosed aspects, embodiments,and/or configurations after understanding the present disclosure. Thepresent disclosure, in various aspects, embodiments, and/orconfigurations, includes providing devices and processes in the absenceof items not depicted and/or described herein or in various aspects,embodiments, and/or configurations hereof, including in the absence ofsuch items as may have been used in previous devices or processes, e.g.,for improving performance, achieving ease and\or reducing cost ofimplementation.

The foregoing discussion has been presented for purposes of illustrationand description. The foregoing is not intended to limit the disclosureto the form or forms disclosed herein. In the foregoing DetailedDescription for example, various features of the disclosure are groupedtogether in one or more aspects, embodiments, and/or configurations forthe purpose of streamlining the disclosure. The features of the aspects,embodiments, and/or configurations of the disclosure may be combined inalternate aspects, embodiments, and/or configurations other than thosediscussed above. This method of disclosure is not to be interpreted asreflecting an intention that the claims require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive aspects lie in less than all features of a singleforegoing disclosed aspect, embodiment, and/or configuration. Thus, thefollowing claims are hereby incorporated into this Detailed Description,with each claim standing on its own as a separate preferred embodimentof the disclosure.

Moreover, though the description has included description of one or moreaspects, embodiments, and/or configurations and certain variations andmodifications, other variations, combinations, and modifications arewithin the scope of the disclosure, e.g., as may be within the skill andknowledge of those in the art, after understanding the presentdisclosure. It is intended to obtain rights which include alternativeaspects, embodiments, and/or configurations to the extent permitted,including alternate, interchangeable and/or equivalent structures,functions, ranges or steps to those claimed, whether or not suchalternate, interchangeable and/or equivalent structures, functions,ranges or steps are disclosed herein, and without intending to publiclydedicate any patentable subject matter. The phrases “at least one,” “oneor more,” and “and/or” are open-ended expressions that are bothconjunctive and disjunctive in operation. For example, each of theexpressions “at least one of A, B and C,” “at least one of A, B, or C,”“one or more of A, B, and C,” “one or more of A, B, or C” and “A, B,and/or C” means A alone, B alone, C alone, A and B together, A and Ctogether, B and C together, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more” and “at least one” can beused interchangeably herein. It is also to be noted that the terms“comprising,” “including,” and “having” can be used interchangeably.

The term “automatic” and variations thereof, as used herein, refers toany process or operation done without material human input when theprocess or operation is performed. However, a process or operation canbe automatic, even though performance of the process or operation usesmaterial or immaterial human input, if the input is received beforeperformance of the process or operation. Human input is deemed to bematerial if such input influences how the process or operation will beperformed. Human input that consents to the performance of the processor operation is not deemed to be “material.”

The term “computer-readable medium” as used herein refers to anytangible storage and/or transmission medium that participate inproviding instructions to a processor for execution. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, NVRAM, or magnetic or optical disks. Volatile media includesdynamic memory, such as main memory. Common forms of computer-readablemedia include, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, magneto-optical medium, aCD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, a solid state medium like a memory card, any other memorychip or cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read. A digital file attachment toe-mail or other self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. When the computer-readable media is configured as a database, itis to be understood that the database may be any type of database, suchas relational, hierarchical, object-oriented, and/or the like.Accordingly, the disclosure is considered to include a tangible storagemedium or distribution medium and prior art-recognized equivalents andsuccessor media, in which the software implementations of the presentdisclosure are stored.

The term “module” as used herein refers to any known or later developedhardware, software, firmware, artificial intelligence, fuzzy logic, orcombination of hardware and software that is capable of performing thefunctionality associated with that element.

The terms “determine,” “calculate,” and “compute,” and variationsthereof, as used herein, are used interchangeably and include any typeof methodology, process, mathematical operation or technique.

Examples of the processors as described herein may include, but are notlimited to, at least one of Qualcomm® Snapdragon® 800 and 801, Qualcomm®Snapdragon® 610 and 615 with 4G LTE Integration and 64-bit computing,Apple® A7 processor with 64-bit architecture, Apple® M7 motioncoprocessors, Samsung® Exynos® series, the Intel® Corem family ofprocessors, the Intel® Xeon® family of processors, the Intel® Atom™family of processors, the Intel Itanium® family of processors, Intel®Core® i5-4670K and i7-4770K 22 nm Haswell, Intel® Core® i5-3570K 22 nmIvy Bridge, the AMD® FX™ family of processors, AMD® FX-4300, FX-6300,and FX-8350 32 nm Vishera, AMD® Kaveri processors, Texas Instruments®Jacinto C6000™ automotive infotainment processors, Texas Instruments®OMAP™ automotive-grade mobile processors, ARM® Cortex™-M processors,ARM® Cortex-A and ARM926FJ-S™ processors, other industry-equivalentprocessors, and may perform computational functions using any known orfuture-developed standard, instruction set, libraries, and/orarchitecture.

It shall be understood that the term “means” as used herein shall begiven its broadest possible interpretation in accordance with 35 U.S.C.,Section 112, Paragraph 6. Accordingly, a claim incorporating the term“means” shall cover all structures, materials, or acts set forth herein,and all of the equivalents thereof. Further, the structures, materialsor acts and the equivalents thereof shall include all those described inthe summary of the invention, brief description of the drawings,detailed description, abstract, and claims themselves.

It is therefore apparent that there has been provided, in accordancewith embodiments of the present invention, systems, apparatuses, andmethods for enhancing call preservation techniques. While this inventionhas been described in conjunction with a number of embodiments, it isevident that many alternatives, modifications, and variations would beor are apparent to those of ordinary skill in the applicable arts.Accordingly, it is intended to embrace all such alternatives,modifications, equivalents, and variations that are within the spiritand scope of this invention.

1. A method of generating a virtual three-dimensional model of anobject, the method comprising: obtaining a plurality of two-dimensionalimages of the object in succession, the plurality of two-dimensionalimages being captured form different orientations; storing saidplurality of two-dimensional images into one or more memories;determining a relative change in position of said plurality oftwo-dimensional images by comparing two subsequent images of theplurality of two-dimensional images, wherein the relative change isdetermined by a difference in color intensity values between pixels ofone of the two subsequent images and another of the two subsequentimages; generating a three-dimensional coordinate based on the relativechange in position; and generating a computer image including thethree-dimensional coordinate, wherein said computer image represents thevirtual three-dimensional model of the object. 2-20. (canceled)